Comment on "Recurrent neural networks: A constructive algorithm, and its properties"

In their paper [1], Tsoi and Tan present what they call a "canonical form", which they claim to be identical to that proposed in Nerrand et al [2]. They also claim that the algorithm which they present can be applied to any recurrent neural network. In the present comment, we disprove both claims. Back in 1993, Nerrand et al. [2] proposed a general approach to the training of recurrent networks, either adaptively (on-line) or non-adaptively (off-line). One of the main points of that paper was the introduction of the minimal state-space form, or canonical form, defined in relations (4) and (4a) of their paper as: z(n+1) = φ [ z(n), u(n)] (state equation) y(n+1) = ψ[ z(n+1), u(n+1)] (output equation) where z(n) is a state vector, i.e. a minimal set of variables necessary at time n for computing the future output vector y(n+1), the external input vector u(n+1) (control inputs, measured disturbances, ...) being known. A graphic representation of the canonical form is shown in Figure 1, assuming that functions φ and ψ are computed by a single neural network. In appendix (1) of their paper, Nerrand et al. showed how to compute the order of any recurrent network, and, in appendix (2) they derived the canonical form of various recurrent network architectures, which had been proposed by other authors. Since then, researchers of the same group made use of the canonical form in various circumstances ([3], [4], [5]). They derived a general proof of existence of the canonical form for a class of nonlinear discrete-time models including neural networks, and they provided a systematic procedure for deriving their canonical form, which was presented and published on various occasions ([6], [7]).